Papers by Chae Won Kim
TroL: Traversal of Layers for Large Language and Vision Models (2024.emnlp-main)
Copied to clipboard
| Challenge: | Existing open-source LLVMs that perform comparably to closed-source models such as GPT-4V are often considered too large, having a larger number of layers. |
| Approach: | They propose a new efficient LLVM family with 1.8B, 3.8B, and 7B LLM model sizes, Traversal of Layers, which enables the reuse of layers in a token-wise manner. |
| Outcome: | The proposed model outperforms open-source models with larger model sizes and outperformed closed-source LLVMs with substantial models. |
CoLLaVO: Crayon Large Language and Vision mOdel (2024.findings-acl)
Copied to clipboard
| Challenge: | Existing Large Language Models (LLMs) and instruction tuning have been used to drive the evolution of Vision Language Model (VLM) towards a versatile general-purpose model. |
| Approach: | They propose a learning strategy of Dual QLoRA to preserve object-level image understanding without forgetting it during visual instruction tuning, thereby achieving a significant leap in numerous VL benchmarks in a zero-shot setting. |
| Outcome: | The proposed model outperforms closed-source models on vision language tasks and achieves a significant leap in numerous benchmarks. |